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^  G  A  O 

Accountability  *  Integrity  *  Reliability 

United  States  General  Accounting  Office 
Washington,  DC  20548 


December  21, 2001 


The  Honorable  Stephen  A.  Perry 
Administrator 

General  Services  Administration 
1800  F  Street  NW 
Washington,  DC  20405 


Subject:  GSA  On-line  Procurement  Programs  Lack  Documentation  and  Reliabilitv  Testing 


Dear  Mr.  Perry: 


Earlier  this  year,  we  conducted  a  review*  of  small  business  participation  in  selected  on-line 
procurement  programs  in  fiscal  years  1999  and  2000.  During  the  course  of  this  review,  we 
compared  the  small  business  share  of  procurement  dollars  awarded  by  two  GSA  on-line 
procurement  programs,  GSA  Advantage!  and  Information  Technology  Solutions  Shop  (ITSS), 
to  the  small  business  share  of  aU  federal  contract  dollars  awarded  and  we  assessed  the 
reliability  of  the  data  needed  for  our  report.^  We  evaluated  the  data  by  (1)  performing 
electronic  tests  of  relevant  fields  and  (2)  requesting  and  reviewing,  if  available,  related 
program  and  system  design  documentation,  audit  and  system  reviews,  and  reports.  We 
determined  that  the  data  were  reliable  enough  for  the  purposes  of  that  report.  However,  we 
did  identify  several  errors  that  GSA  needed  to  correct  to  more  accurately  state  total  sales  and 
small  business  participation.  We  also  identified  serious  management  control  weaknesses 
related  to  data  reliability  as  well  as  GSA’s  ability  to  ensure  the  continuity  of  the  operations 
for  the  on-line  programs.  This  letter  discusses  our  concerns  and  recommends  improvement 
actions. 


BACKGROUND 


The  federal  government  is  taking  steps  to  increase  its  use  of  electronic  commerce, 
particularly  in  the  area  of  conducting  on-line  procurement.  For  example,  as  part  of  an  effort 
to  improve  the  functioning  of  the  government  and  achieve  efficiencies  of  operations,  the 
President  has  requested  agencies  to  expand  their  use  of  on-line  procurement  in  fiscal 
year  2002.' 


‘  Electronic  Commerce:  Small  Business  PartiiCi/nation  in  Selected  On-liru!  Procurement  Pmamms  (GAO-02-1,  Oct  29,  2001). 
‘  Generally  accepted  government  auditing  standards  require  that  we  assess  the  reliability  of  any  data  used  to  support  our 
findings,  conclusions,  and  recommendations. 

®  0MB  Memorandum  No.  01-11,  Feb.  14, 2001. 
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GSA  Advantage!  and  ITSS  are  two  systems  agencies  can  turn  to  for  electronic  procurements. 
GSA  Advantage!  serves  as  an  “electronic  mall”  in  which  government  buyers  can  search 
listings,  compare  prices,  and  purchase  items  on-line.  ITSS  on-line  procurement  program  is 
used  for  larger,  complex  purchases.  Customers  enter  requirements  on-line,  GSA  staff  solicit 
quotes,  issue  purchase  orders,  and  authorize  payment  electronically.  GSA  administers  both 
Advantage!  and  ITSS  on-line  procurement  programs.  Sales  through  these  two  on-line 
programs  have  been  small  but  growing.  The  two  on-line  programs  grew  as  a  percentage  of 
total  federal  procurement  dollars  from  about  one-half  percent  in  fiscal  year  1999  to  about 
1  percent  in  fiscal  year  2000.  The  table  below  provides  a  description  of  each  of  the  programs. 


Table  1 :  GSA’s  Advantage!  and  ITSS  Programs:  Services.  Products,  and  Sales 

Dollars  in  millions 


Fiscal  year  sales 

Program 

Services 

Products 

(Actual)  1999 

(Actual)  2000 

Advantage! 

Web-based  catalog 
ordering  system 

Over  2  million  commercial 
products  and  services,  many 
of  which  are  office  supplies 
and  Information  technology 
products  and  services 

$85,687 

$124,892 

ITSS 

Web-based 
contracting  services 

Information  technology 
products  and  services,  from 
computer  monitors  to 
complete  network  installations 

977,472 

2,258,709 

Source:  GAO,  based  on  GSA  documentation 


It  is  important  that  the  data  in  both  systems  be  as  accurate  and  reliable  as  possible. 
Government  agencies,  for  example,  need  to  know  that  pricing  data  is  accurate  to  make  their 
purchases.  Agencies  also  rely  on  these  systems  for  summary  data  on  government  contracting 
opportunities,  solicitations,  awards,  etc.  The  Congress  and  other  oversight  entities  need  to 
know  that  data  on  such  things  as  total  sales,  agency  use,  and  small  business  participation  is 
reliable  in  order  to  assess  the  effectiveness  of  government  contracting  programs. 


LACK  OF  SYSTEM  DOCUMENTATION 


GSA  program  officials  told  us  that  they  did  not  have  system  documentation  for  either  GSA 
Advantage!  or  ITSS.  Documentation  would  normally  include  the  data  processing  or 
automated  aspects  of  the  system  (such  as  application  program  coding),  operating 
instructions,  and  system  policies,  processes,  and  procedures  (such  as  procedures  on 
“configuration  control”  or  how  changes  to  the  system  are  to  be  made).  Documenting  systems 
is  a  common  management  practice  and  is  required  of  GSA  under  the  Joint  Financial 
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Management  Improvement  Program’s^  Federal  Financial  System  Requirements  to  completely 
document  both  management  systems. 


According  to  GSA  Advantage!  officials,  there  is  no  formal  documentation  of  the  system 
because  the  system  evolved  over  many  years  in  a  piecemeal  fashion  among  many  developers 
-both  government  and  contractors.  TTie  Chief  Technology  Officer  for  ITSS  also  said  that 
rrSS  was  developed  without  extensive  documentation  and  evolved  over  time,  based  on  rapid 
prototyping  and  deployment.  GSA  officials  told  us  that  they  have  just  started  efforts  to 
contract  for  services  to  document  the  systems. 


It  is  important  for  GSA  to  take  time  now  to  document  the  overall  system.  Documenting 
systems  helps  organizations  ensure  the  data  is  reliably  collected  and  entered  into  a  system 
and  that  changes  made  are  appropriate  and  completed  in  a  timely  and  effective  manner. 
Moreover,  by  documenting  such  things  as  coding  and  system  operations,  organizations  are 
better  positioned  to  ensure  that  they  can  continue  operations  in  the  event  of  a  disaster  or 
emergency.  This  is  especially  important  for  GSA  because  it  relies  heavily  on  contractors  to 
operate  and  maintain  the  two  systems. 


LACK  OF  DATA  ENTRY  CONTROLS  AND  ROUTINE  DATA  VALIDATION 


As  part  of  our  reliability  assessment,  we  electronically  analyzed  data  in  both  GSA  Advantage! 
and  ITSS  and  found  significant  errors.  For  example: 


•  In  liie  case  of  Advantage!,  the  database  included  $32.2  million  that  represented  16  “test 
orders”,  which  were  used  to  train  and  provide  demonstrations  to  users,  according  to  GSA 
officials.  These  data  were  not  marked  as  test  cases  and  their  inclusion  would  have 
skewed  future  reports,  including  small  business  participation  and  data  extracts. 

Currently,  there  is  no  separate  training  environment  to  ensure  that  test  orders  are  not 
included  with  real  orders. 


•  In  the  case  of  ITSS,  we  foimd  that  GSA  was  using  an  incorrect  formula  to  calculate  total 
sales.  As  a  result,  initially  fiscal  year  sales  totaled  $3.1  billion;  as  corrected,  they  summed 
to  $2.3  billion.  The  ITSS  Chief  Technology  Officer  told  us  that  ITSS  was  not  originally 
designed  to  provide  strong  reporting  capability-  Consequently,  those  trying  to  generate 
reports  from  the  system  need  a  thorough  understanding  of  the  data  and  data  queries  to 
obtain  correct  information. 


*  The  Joint  Financial  Management  Improvement  Program  (JFMIP)  is  a  joint  undertaking  of  the  US  Department  of  Treasury,  the 
General  Accounting  Office,  the  Office  of  Management  and  Budget  and  the  Office  of  Personnel  Management,  working  in 
cooperation  with  each  other  and  other  agencies  to  improve  financial  management  practices  in  Government.  JFMIP’s  guidance 
covers  acquisition  systems  such  as  Advantage!  and  ITSS. 
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In  each  case,  contract  and  program  officials  agreed  with  our  analysis  and  subsequently 
corrected  the  databases  and/or  reported  numbers  to  reflect  more  accurate  numbers. 


The  data  problems  we  identified  may  be  attributable  to  the  fact  that  GSA  lacks  data  entry  or 
edit  controls  to  catch  this  type  of  errors  and  does  not  periodically  validate  data  Such 
controls  are  considered  to  be  common  industry  practice  since  they  help  ensure  that  data  are 
accurate  and  reliable.  Although  contractors  told  us  that  the  system  was  designed  to  “police 
itself’ — ^that  is,  its  customers,  contracting  officials  and  other  agencies  would  point  out 
errors — ^the  errors  we  discovered  had  not  previously  been  brought  to  GSA’s  attention. 
Moreover,  GSA  officials  did  not  affirm  that  they  would  be  implementing  routine  data 
validation  and  other  safeguards  to  ensure  data  accuracy  and  integrity. 


CONCLUSION 


GSA’s  two  online  procurement  systems — GSA  Advantage!  and  ITSS — lack  basic  management 
controls  to  ensure  the  systems  produce  reliable  data,  can  not  be  inappropriately  modified, 
and  provide  continuous  service.  These  controls  are  increasingly  important  since  government 
agencies  eire  expected  to  increase  their  use  of  these  systems  and  GSA  relies  heavily  on 
contractors  for  systems  operations  and  maintenance.  Moreover,  the  Congress  and  other 
oversight  entities  need  adequate  assurance  that  these  systems  are  reliable  and  accurate  to 
assess  the  effectiveness  of  government  contracting  programs. 


RECOMMENDATIONS  FOR  AGENCY  ACHON 


We  recommend  that  the  GSA  Administrator  direct  managers  responsible  for  oversight  of  both 
GSA  Advantage!  and  ITSS  to: 


•  Fully  document  the  procurement  systems  in  accordance  with  guidelmes  for  government 
financial  management  systems,  such  as  those  in  JFMIP; 


•  Establish  procedures  and  deadlines  for  documenting  any  future  changes  to  either  system; 

•  Establish  procedures  to  prevent  entry  of  test  data  into  production  systems;  and 

•  Develop  procedures  for  periodic  examination,  validation,  and  timely  correction  of  GSA 
Advantage!  and  ITSS  data. 
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AGENCY  COMMENTS 


We  provided  GSA  with  a  draft  of  this  report  for  review  and  comments.  In  oral  comments, 
GSA  officials  agreed  with  our  recommendations  and  provided  technical  comments,  which  we 
have  incorporated  as  appropriate.  GSA  officials  also  said  that  even  though  both  systems  are 
being  reengineered,  the  issues  we  raised  continue  to  be  relevant,  and  that  they  are  initiating 
actions  to  address  our  recommendations.  The  officials  could  not  estimate  when 
implementation  of  the  recommendations  would  be  complete. 


To  evaluate  the  reliability  of  data  provided  by  contractors  and  officials  responsible  for 
Advantage!  and  ITSS,  we: 


•  reviewed  govemmentwide  guidance  for  developing  and  maintaining  management 
systems; 


•  reviewed  existing  agency  information  such  as  audit  reports  by  the  GSA  Inspector 
General,  independent  reviews  and  system  documentation,  if  available; 


•  performed  electronic  data  testing  of  relevant  fields  such  as  checking  for  missing  data,  the 
relationship  of  one  field  to  another,  values  outside  of  a  designated  range,  and  dates 
outside  valid  time  flmnes;  and 


•  discussed  control  weaknesses  with  officials  and  contractors. 


We  conducted  our  review  from  April  2001  to  September  2001  in  accordance  with  generally 
accepted  government  auditing  standards.  In  conducting  our  review  we  met  with  officials  at 
the  General  Services  Administration,  and  contractors  for  both  Advantage!  and  ITSS  on-line 
procurement  programs. 


This  report  contains  recommendations  to  you.  The  head  of  a  federal  agency  is  required 
under  31  U.S.C.  720  to  submit  a  written  statement  of  actions  taken  on  our  recommendations 
to  the  Senate  Committee  on  Governmental  Affairs  and  the  House  Committee  on  Government 
Reform  not  later  than  60  days  after  the  date  of  this  letter  and  to  the  Senate  and  House 
Committees  on  Appropriations  with  the  agency’s  first  request  for  £q)propriations  made  more 
than  60  days  after  the  date  of  this  letter. 
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We  are  sending  copies  of  this  letter  to  interested  congressional  committees.  This  letter  will 
also  be  available  on  GAO’s  home  page  at  http://www.  gao.gov.  Please  contact  me  at 
(202)  512-4841  or  Hilary  Sullivan  at  (214)  777-5652  if  you  or  your  staff  have  questions.  Major 
contributors  to  this  report  were  Barbara  Johnson,  Rosa  M.  Johnson,  and  Beverly  Ross. 


Sincerely  yours. 


David  E.  Cooper 
Director,  Acquisition 
and  Sourcing  Management 


(120106) 
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